feat(attachments, storage): u#3727 t#3961 physical attachment deletion system #476

bameda · 2023-09-04T08:34:02Z

This PR ensures that the files are deleted when the attachments are deleted. Several things must be taken into consideration:

Currently, files are saved on disk, but this will change so we will have to use django's storage module.
In the future we will have to postpone the deletion and make it easily recoverable (to avoid data loss). This is in line with the logical deletion of objects.

The solution that has been reached consists of:

A new module taiga.commons.storage where everything related to file storage will be created.
A new StoragedObject model that will store the information about where the file is stored.
Now the Attachment model will have a ForeingKey to StoragedObject
A signal is defined for Attachment during the post_save that will mark the associated StoragedObject as deleted.
A periodic task is created that will physically delete all files that belong to StoragedObject records that have been deleted more than X days ago. (The frequency of the periodic task and time that the files will be kept are configurable parameters).
There is a CLI command to remove deleted StoragedObject items and their files.

How to tests:

Review the code.
Tests are green.
Manual tests: Tests logical deletion:
- Init:
  - 👸 Open the backend admin panel and access to Storaged Objects list view
  - 👤 Open the frontend
- Tests delete attachment:
  - 👤 Go to a story and upload some attachments
  - 👸 refresh and see new storaged objets created
  - 👤 Delete attachments
  - 👸 refresh and see storaged objets with their deleted datetime.
- Tests delete project:
  - 👤 Create a new project.
  - 👤 Create a new story and upload some attachments (you can create more than one story with attachments)
  - 👸 refresh and see new storaged objets created
  - 👤 Delete the project
  - 👸 refresh and see all the storaged objets with their deleted datetime.
- [Optional] Tests delete user with workspaces and projects with stories with atachments:
  - You can test a more complex scenario: deleting a user involves deleting workspaces that have projects with stories "full" of attachments.
Manual tests: Tests physical deletion:
- Test CLI: python -m taiga storage clean-storaged-objects --help
- Test periodic task: You can modify TAIGA_STORAGE__DAYS_TO_STORE_DELETED_STORAGED_OBJECTS (int or None for all) and TAIGA_STORAGE__DAYS_TO_STORE_DELETED_STORAGED_OBJECTS (use cron format)

github-actions · 2023-09-04T08:57:46Z

Coverage Report

File	Stmts	Miss	Cover	Missing
src/taiga
__main__.py	71	71	0%	17–155
wsgi.py	15	15	0%	8–31
src/taiga/attachments
admin.py	27	3	90%	25, 28, 70
events.py	8	2	75%	15, 20
models.py	19	2	89%	60, 63
repositories.py	49	2	93%	62, 64
src/taiga/auth/services
__init__.py	47	0	98%	90–>93
src/taiga/base
mocks.py	74	12	80%	93–94, 107–118, 143–>exit
src/taiga/base/api
permissions.py	49	1	99%	22
requests.py	11	3	80%	15, 21–23
src/taiga/base/db
__init__.py	4	1	75%	15
commands.py	54	54	0%	8–271
sequences.py	45	2	97%	91–92
src/taiga/base/db/admin
forms.py	14	8	38%	17–25
utils.py	11	7	36%	17–23
src/taiga/base/db/models
fields.py	15	1	88%	20
src/taiga/base/django
__init__.py	5	2	60%	17–19
settings.py	30	2	93%	16–17
urls.py	5	5	0%	8–13
wsgi.py	4	4	0%	8–14
src/taiga/base/i18n
__init__.py	102	9	88%	171–>170, 173–>170, 192–193, 203–205, 215, 268, 279, 283
choices.py	34	2	91%	80, 87
commands.py	74	74	0%	8–108
src/taiga/base/logging
formatters.py	24	10	50%	22–>exit, 38, 44–45, 51–60
middlewares.py	23	1	96%	20
src/taiga/base/repositories
neighbors.py	39	3	91%	43–44, 65
src/taiga/base/sampledata
commands.py	11	11	0%	8–40
constants.py	15	15	0%	14–51
demo_data.py	273	273	0%	8–529
factories.py	122	122	0%	9–353
test_data.py	290	290	0%	8–634
src/taiga/base/serializers
fields.py	20	1	96%	21
src/taiga/base/utils
commands.py	31	31	0%	8–60
datetime.py	44	18	60%	54, 67, 126–147
enum.py	14	1	92%	43
images.py	26	3	87%	21–22, 30
json.py	33	15	45%	30–35, 37, 39–44, 46, 50
pprint.py	6	6	0%	8–17
slug.py	23	4	85%	22–25
src/taiga/base/validators/fields
i18n.py	16	2	91%	19–20
uuid.py	15	2	90%	19–20
src/taiga/comments
admin.py	29	3	92%	16, 20, 24
events.py	11	3	73%	15, 20, 25
models.py	20	2	91%	43, 46
src/taiga/commons/storage
commands.py	13	13	0%	8–40
models.py	16	2	88%	38, 41
services.py	9	0	92%	17–>16
tasks.py	9	9	0%	8–19
src/taiga/conf
__init__.py	93	3	95%	109, 125–126
tokens.py	17	1	90%	34
src/taiga/emails
commands.py	35	35	0%	9–68
sender.py	18	5	71%	48–50, 54–73
tasks.py	39	4	90%	44–45, 53–54
src/taiga/events
app.py	18	1	92%	21
events.py	16	2	88%	31–32
logging.py	39	23	31%	18–>exit, 32–48, 51–60
manager.py	117	11	90%	78–79, 223–233, 239–242
responses.py	33	6	84%	33, 37–41
subscriber.py	80	4	95%	51, 55, 88, 91, 108–>120
src/taiga/events/actions
auth.py	23	11	44%	27–32, 39–45
event_handlers.py	9	4	56%	18–20, 28–30
projects.py	55	30	38%	36–40, 48–64, 72–80, 88–96
workspaces.py	55	30	38%	36–40, 48–64, 72–80, 88–96
src/taiga/events/pubsub/backends
base.py	30	1	98%	53
redis.py	66	5	94%	41–43, 85–86
src/taiga/exceptions/api
__init__.py	27	0	97%	28–>30
handlers.py	27	2	90%	27–28
middlewares.py	26	2	87%	37, 41
src/taiga/integrations/auth
services.py	39	7	82%	50–56
src/taiga/integrations/github
services.py	40	0	98%	63–>68
src/taiga/mediafiles
admin.py	20	2	91%	24, 27
models.py	26	2	92%	84, 87
src/taiga/permissions
services.py	103	1	98%	28, 106–>109
src/taiga/permissions/validators
fields.py	24	3	88%	18–19, 45
src/taiga/projects/invitations
models.py	25	2	92%	89, 92
repositories.py	84	1	98%	135
src/taiga/projects/invitations/api
validators.py	65	7	89%	38–40, 65–66, 84–85
src/taiga/projects/invitations/events
__init__.py	33	1	91%	37–>exit, 49–>exit, 74–>exit, 102
src/taiga/projects/invitations/services
__init__.py	183	1	99%	58–>53, 286
src/taiga/projects/memberships
models.py	16	2	88%	51, 54
repositories.py	86	2	97%	91, 124
src/taiga/projects/memberships/events
__init__.py	16	2	88%	18–24
src/taiga/projects/memberships/services
__init__.py	45	1	95%	64–>69, 111
src/taiga/projects/projects
admin.py	53	12	75%	33–34, 39–42, 50–51, 56–59, 90, 94
models.py	60	8	86%	64, 67, 78, 112, 115, 118–120
repositories.py	115	4	96%	125, 126–>129, 155, 254–255
tasks.py	6	1	88%	17
src/taiga/projects/projects/services
__init__.py	105	5	92%	77, 86, 192, 198–>201, 206, 228
src/taiga/projects/references
mixins.py	13	0	93%	25–>28
src/taiga/projects/roles
api.py	35	3	93%	46–48
models.py	26	3	86%	50, 53, 57
repositories.py	44	1	96%	92
src/taiga/projects/roles/services
__init__.py	25	1	94%	24, 57–>62
src/taiga/stories/assignments
models.py	15	2	87%	42, 45
repositories.py	52	1	97%	49
src/taiga/stories/attachments
api.py	42	1	96%	128
src/taiga/stories/comments
api.py	58	1	97%	177
src/taiga/stories/stories
admin.py	25	5	84%	76–81, 85, 89
models.py	25	2	92%	73, 76
src/taiga/stories/stories/services
__init__.py	113	2	97%	233, 256
src/taiga/tasksqueue
app.py	20	2	86%	29–30
commands.py	65	65	0%	8–122
logging.py	33	17	40%	18–>exit, 29–39, 42–51
manager.py	56	13	77%	37, 41, 45, 49, 94, 163, 177, 186, 191, 198, 202, 209, 213
task.py	12	1	92%	23
src/taiga/tokens
admin.py	44	10	81%	67, 70, 73, 76, 104, 108, 112, 116, 120, 124
models.py	30	4	87%	59, 62, 78, 81
src/taiga/users
admin.py	51	7	81%	32, 41, 50, 87–95
commands.py	9	9	0%	8–26
models.py	65	10	86%	71, 74, 90, 93, 97, 100, 103, 106, 127, 130
repositories.py	204	3	98%	209–>212, 335, 428–429
src/taiga/users/api
__init__.py	66	2	96%	216, 304
src/taiga/users/api/validators
__init__.py	42	3	92%	39–41
src/taiga/users/services
__init__.py	244	7	95%	75, 259, 262–>257, 297–>280, 306–>280, 322–>318, 345–>341, 366, 494–>499, 513–520, 521–>526, 524–525
src/taiga/workflows
models.py	36	4	89%	39, 42, 72, 75
src/taiga/workflows/services
__init__.py	91	3	94%	171, 198, 269–>277, 284
src/taiga/workspaces/invitations
models.py	24	2	92%	84, 87
permissions.py	9	1	82%	19
repositories.py	77	1	98%	123
src/taiga/workspaces/invitations/api
validators.py	18	2	88%	24–25
src/taiga/workspaces/invitations/events
__init__.py	25	1	91%	36–>exit, 60–>exit, 71
src/taiga/workspaces/invitations/services
__init__.py	123	1	99%	222
src/taiga/workspaces/memberships
api.py	43	1	96%	126
models.py	15	2	87%	43, 46
repositories.py	63	2	95%	72, 105
src/taiga/workspaces/memberships/services
__init__.py	46	1	97%	117
src/taiga/workspaces/workspaces
admin.py	57	19	62%	25–26, 31–34, 49, 52–57, 61, 65, 73–74, 79–82
models.py	18	2	90%	32, 35
repositories.py	127	2	99%	232–233
src/taiga/workspaces/workspaces/services
__init__.py	54	2	94%	113, 116
TOTAL	9779	1587	82%

Tests	Skipped	Failures	Errors	Time
1125	1 💤	0 ❌	0 🔥	23m 9s ⏱️

python/apps/taiga/src/taiga/attachments/apps.py

python/apps/taiga/src/taiga/commons/storage/models.py

daniel-herrero · 2023-09-08T10:30:46Z

python/apps/taiga/src/taiga/commons/storage/admin.py

+@admin.register(StoragedObject)
+class StoragedObjectAdmin(admin.ModelAdmin[StoragedObject]):
+    list_display = (
+        "id" "file",


You missed a comma

…n system

daniel-herrero

Mega clap!

bameda marked this pull request as draft September 4, 2023 08:36

bameda force-pushed the us/3727/back_clean_deleted_attachments branch 8 times, most recently from 8841c36 to 67b776c Compare September 5, 2023 16:09

bameda changed the title ~~feat(attachments, store): u#3727 t#3961 physical attachment deletion system.~~ feat(attachments, store): u#3727 t#3961 physical attachment file deletion system. Sep 5, 2023

bameda force-pushed the us/3727/back_clean_deleted_attachments branch 4 times, most recently from a226a7a to 449e660 Compare September 6, 2023 18:37

bameda marked this pull request as ready for review September 6, 2023 18:41

bameda force-pushed the us/3727/back_clean_deleted_attachments branch from 449e660 to d37d442 Compare September 6, 2023 19:34

bameda changed the title ~~feat(attachments, store): u#3727 t#3961 physical attachment file deletion system.~~ feat(attachments, storage): u#3727 t#3961 physical attachment deletion system Sep 6, 2023

bameda force-pushed the us/3727/back_clean_deleted_attachments branch 2 times, most recently from d772072 to ff3f953 Compare September 6, 2023 21:07

daniel-herrero self-assigned this Sep 7, 2023

daniel-herrero reviewed Sep 7, 2023

View reviewed changes

python/apps/taiga/src/taiga/attachments/apps.py Show resolved Hide resolved

daniel-herrero reviewed Sep 7, 2023

View reviewed changes

python/apps/taiga/src/taiga/commons/storage/models.py Outdated Show resolved Hide resolved

bameda force-pushed the us/3727/back_clean_deleted_attachments branch 2 times, most recently from 94bb974 to dde6bf1 Compare September 8, 2023 09:58

daniel-herrero reviewed Sep 8, 2023

View reviewed changes

feat(attachments, storage): u#3727 t#3961 physical attachment deletio…

3547390

…n system

bameda force-pushed the us/3727/back_clean_deleted_attachments branch from dde6bf1 to 3547390 Compare September 8, 2023 10:37

daniel-herrero approved these changes Sep 8, 2023

View reviewed changes

bameda merged commit 9028144 into main Sep 8, 2023
2 checks passed

bameda deleted the us/3727/back_clean_deleted_attachments branch September 8, 2023 11:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(attachments, storage): u#3727 t#3961 physical attachment deletion system #476

feat(attachments, storage): u#3727 t#3961 physical attachment deletion system #476

bameda commented Sep 4, 2023 •

edited by daniel-herrero

Loading

github-actions bot commented Sep 4, 2023 •

edited

Loading

daniel-herrero Sep 8, 2023

bameda Sep 8, 2023

daniel-herrero left a comment

feat(attachments, storage): u#3727 t#3961 physical attachment deletion system #476

feat(attachments, storage): u#3727 t#3961 physical attachment deletion system #476

Conversation

bameda commented Sep 4, 2023 • edited by daniel-herrero Loading

How to tests:

github-actions bot commented Sep 4, 2023 • edited Loading

daniel-herrero Sep 8, 2023

Choose a reason for hiding this comment

bameda Sep 8, 2023

Choose a reason for hiding this comment

daniel-herrero left a comment

Choose a reason for hiding this comment

bameda commented Sep 4, 2023 •

edited by daniel-herrero

Loading

github-actions bot commented Sep 4, 2023 •

edited

Loading